Skip to content

feat(responses): add opt-in hosted Responses continuation state - #59

Merged
ALikesToCode merged 2 commits into
feat/gateway-batchesfrom
feat/responses-state
Oct 9, 2026
Merged

ALikesToCode merged 2 commits into
feat/gateway-batchesfrom
feat/responses-state

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds opt-in hosted Responses continuation state. With HOSTED_RESPONSES_ENABLED off (the default), /v1/responses requests, response bytes, headers and storage are unchanged. That includes store, previous_response_id and a gateway_state field. GET and DELETE /v1/responses/<id> keep their existing routing.

  • Opt-in per request:
    • On the managed POST /v1/responses route, gateway_state: true together with the literal store: true asks the gateway to keep the state.
    • gateway_state is removed before dispatch, and provider-side storage is turned off for that generation.
    • A completed response gets a gateway ID: gwresp_ plus 32 hex characters.
  • Continuation:
    • previous_response_id with a gateway ID loads the caller's stored state. It rebuilds the original input, the completed output items and the new input in order, keeping tool call/result boundaries exact.
    • Absent, expired, malformed or foreign IDs return JSON 404 with no provider call.
    • A gateway parent without gateway_state runs once with the retained context and stores nothing new.
  • Provider parents:
    • Provider-issued IDs keep today's behaviour.
    • Combining hosted opt-in with a provider-owned parent returns 400, because the gateway can't retain history it never saw. This is a deliberate deviation from "unchanged".
  • Read and delete:
    • GET and DELETE /v1/responses/<id> act only on gateway IDs owned by the caller (404 otherwise), with Cache-Control: no-store.
    • DELETE removes the D1 row and the R2 body. TTL expiry is the only automatic deletion.
  • What gets stored:
    • Only a completed terminal status with settled accounting, where usage is measured or explicitly unknown.
    • Failed, incomplete, cancelled, deadline-expired and ambiguous outcomes can't be read or continued.
    • An unverified completion returns response_outcome_unknown without dispatching again. Storage errors return 503 and never trigger a provider retry.
  • Streaming:
    • State is stored after the terminal response.completed event, a clean end of stream and accounting settlement. The terminal event is held until storage succeeds.
    • An interrupted stream stores nothing. There is no SSE reattachment.
  • Idempotency: with managed idempotency on, a keyed non-streaming request replays the same gateway ID. An unknown earlier outcome stays blocked.
  • Limits: 1 MiB per stored state, a 24-hour TTL and a chain depth of 16. Oversized input or a deeper chain returns 400 before dispatch.
  • Retention: zero-content retention returns 400 retention_conflict before dispatch for hosted opt-in and gateway continuation. It also blocks retrieval, but explicit deletion still works.
  • Storage:
    • D1 hosted_responses (migration 0028) holds the owner, status, provider/model, parent, policy revision, expiry, body pointer, length and SHA-256.
    • Bodies are under responses-state/ in the existing multillm_media bucket.
    • Flask reaches them through the private /v1/managed-state/responses operation.
    • With the flag on and storage missing, a hosted request gets JSON 503 before any provider call.

Details are in docs/responses-state.md.

Wiring

  • services/gateway_extensions.py:
    • registers the hosted Responses hook right after retention and before the deadline hook;
    • defers the idempotency finalizer for hosted turns, so completion settles after accounting and the deadline checks.
  • routes/unified.py: dispatch_protocol_request gains one decorator, @with_responses_state. The file is now exactly 1,000 lines.
  • services/protocol_translation/responses.py: response_object gets one small call site that rewrites the ID for hosted turns. It returns the same object otherwise.
  • worker/managed-state-dispatch.mjs routes /v1/managed-state/responses to handleResponsesStateRequest.

Shared registrations

  • tests/test_responses_state_worker.mjs in test:worker.
  • worker/responses-state-d1.mjs in the edge module inventory; app.route('/v1/responses/<response_id>', methods=['GET', 'DELETE']) in the Flask route inventory.
  • HOSTED_RESPONSES_ENABLED in the Container env keys.
  • hosted_responses in REQUIRED_D1_TABLES.
  • tests/test_wave4_registrar.py now expects responses_state_hook between the retention and deadline hooks.
  • The migration was renumbered from 0026 to 0028 to follow the Wave 5 stack.

Stacked on #57

Migration numbers must be contiguous in each PR, so this branch stacks on feat/gateway-batches (#57, migration 0027). That one stacks on #56, #54 and #50. Merge those first.

Tests

  • tests/test_responses_state.py and tests/test_responses_state_worker.mjs cover:
    • flag off, byte for byte, including GET and DELETE routing;
    • a malformed flag warning without its value;
    • store, get and continue preserving tool item order, including parallel tool calls;
    • absent, foreign and expired parents returning 404 with no provider call;
    • explicit deletion;
    • zero retention refusing before storage;
    • storage failures that are visible and never retried;
    • non-terminal, failed, lost and ambiguous outcomes storing nothing;
    • chain depth and size limits;
    • streams stored only after the terminal event and accounting, and nothing stored on interruption or client abort;
    • keyed replay of the gateway ID and a concurrent duplicate claim allowing one generation;
    • a deadline expiring during a storage acknowledgement;
    • D1/R2 races (expiry during a write or CAS, deletion racing a late body);
    • an additive, re-entrant migration.
  • Full Python suite: 4,142 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,560 passed. npm run test:integration: 7 passed.
  • migrations:check (28 contiguous, additive migrations), test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

Apply the migration before enabling the flag: npx wrangler d1 migrations apply multillm-intelligence --remote.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 5. The README line comes with the Wave 5 integration PR.

With HOSTED_RESPONSES_ENABLED on, a managed /v1/responses request with gateway_state and store keeps the completed response under a gateway-issued ID. previous_response_id continues from it in order, and GET and DELETE act only on the caller's own IDs. State lives in D1 and R2 with a 24-hour TTL, and only classified complete successes are stored.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy a4630c5 Oct 09 2026, 02:58 AM

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1b46e00b-98e5-4261-89e7-2035b797e799

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ALikesToCode
ALikesToCode merged commit f2283ba into feat/gateway-batches Oct 9, 2026
4 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the feat/responses-state branch October 26, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant